Skip to content

feat: image capabilities — upload, screenshot passthrough, vision detection & fallback, aria refs#6682

Merged
TimothyZhang7 merged 12 commits into
mainfrom
feat/image-capabilities
Mar 21, 2026
Merged

feat: image capabilities — upload, screenshot passthrough, vision detection & fallback, aria refs#6682
TimothyZhang7 merged 12 commits into
mainfrom
feat/image-capabilities

Conversation

@TimothyZhang7

@TimothyZhang7 TimothyZhang7 commented Mar 21, 2026

Copy link
Copy Markdown
Collaborator

Summary

End-to-end image/vision support across the stack, plus a stable browser element ref system for browser automation:

  • User image upload: attach images to chat messages from the UI; forwarded to the LLM as image content blocks.
  • GCU screenshot passthrough: browser tool screenshots are forwarded to the LLM as image content alongside the text result.
  • Vision capability detection: new capabilities.py module checks whether a model supports images (allow/deny lists per provider and model name).
  • Automatic image stripping: image blocks are stripped before non-vision model calls to prevent API errors.
  • Vision fallback: when the primary model lacks vision, a fallback vision model describes images as text and injects the description.
  • Aria ref system: refs.py annotates aria_snapshot() output with [ref=eN] markers so the LLM can target browser elements by stable ID instead of fragile CSS selectors.

Closes

Test plan

  • make test passes (unit tests for capabilities, refs, screenshot normalization)
  • Upload an image in the chat UI — verify it reaches the LLM
  • Run a browser automation task with a vision-capable model — verify screenshots are forwarded
  • Configure a text-only model (e.g. DeepSeek-R1) — verify images are stripped/described without API errors
  • Use selector="eN" ref in a browser tool call — verify it resolves correctly

@github-actions

Copy link
Copy Markdown

PR Requirements Warning

This PR does not meet the contribution requirements.
If the issue is not fixed within ~24 hours, it may be automatically closed.

PR Author: @TimothyZhang7
Found issues: #6579 (assignees: none), #6678 (assignees: none), #6679 (assignees: none), #6680 (assignees: none), #6681 (assignees: none)
Problem: The PR author must be assigned to the linked issue.

To fix:

  1. Assign yourself (@TimothyZhang7) to one of the linked issues
  2. Re-open this PR

Exception: To bypass this requirement, you can:

  • Add the micro-fix label or include micro-fix in your PR title for trivial fixes
  • Add the documentation label or include doc/docs in your PR title for documentation changes

Micro-fix requirements (must meet ALL):

Qualifies Disqualifies
< 20 lines changed Any functional bug fix
Typos & Documentation & Linting Refactoring for "clean code"
No logic/API/DB changes New features (even tiny ones)

Why is this required? See #472 for details.

@github-actions github-actions Bot added pr-requirements-warning PR doesn't follow contribution guidelines. Please fix or it will be auto-closed. and removed pr-requirements-warning PR doesn't follow contribution guidelines. Please fix or it will be auto-closed. labels Mar 21, 2026
@TimothyZhang7
TimothyZhang7 merged commit 3e1282b into main Mar 21, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment